The Conversion of Diagrams to Knowledge Bases

If future electronic documents are to be truly useful, we must devise ways to automatically turn them into knowledge bases. In particular, we must be able to do this for diagrams. This paper discusses biological diagrams. We describe the three major aspects of diagrams: visual salience, domain conventions and pragmatics. We next describe the organization of diagrams into informational and substrate components. The latter are typically collections of objects related by Generalized Equivalence Relations. To analyze diagrams, we define Graphics Constraint Grammars (GCGs) that can be used for both syntactic and semantic analysis. Each grammar rule describes a rule object and consists of a Production, describing the constituents of the object, Constraints that must hold between the constituents and Propagators that build properties of the rule object from the constituents. We discuss how a mix of Darsing and constraint satisfaction 2. A biological diagram example The example chosen for this discussion is the gene diagram in Figure 1, based on a diagram in [7]. Though x,y data graphs are the most common diagram in all scientific publications, we have already discussed them elsewhere [I-31. Figure 1 shows three related gene segments, in schematic form. The figure includes an expansion of a portion of a segment to the level of the individual DNA bases, A, T, G and C. 3. Diagram organization The effectiveness of diagrams is dependent on three factors: visual salience, domain conventions and pragmatics. These are best understood from the examples below, keyed to Figure 1. Understanding these three organizational principles is necessary if we are to write techniques are used to parse Gagrams with GCGs. grkmars and parsers to analyze diagrams.